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Abstract 



My purpose here is to, once and for all, rid the world of the misconception 
that a pixel is a little geometric square. This is not a religious issue. This is an is- 
sue that strikes right at the root of correct image (sprite) computing and the abil- 
ity to correctly integrate (converge) the discrete and the continuous. The little 
square model is simply incorrect. It harms. It gets in the way. If you find yourself 
thinking that a pixel is a little square, please read this paper. I will have suc- 
ceeded if you at least understand that you are using the model and why it is 
permissible in your case to do so (is it?). 

Everything I say about little squares and pixelsinthe2D case applies equally 
well to little cubes and voxels in 3D. The generalization is straightforward, so I 
won't mention it from hereom. 

I discuss why the little square model continues to dominate our collective 
minds. I show why it is wrong in general. I show when it is appropriate to use a 
little square in the context of a pixel. I propose a discrete to continuous map- 
ping— because this is where the problem arises— that always works and does not 
assume too much. 

I presented some of this argument in Tech Memo 5 ([Smith95]) but have en- 
countered a serious enough misuse of the little square model since I wrote that 
paper to make me bel ieve a ful I frontal attack is necessary. 



The Littie Square Mo6e\ 



The little square model pretends to represents a pixel (picture element) as a 
geometric square2. Thus pixel (i, j) is assumed to correspond to the area of the 
plane bounded by the square ■Kx,y) | i-.5<x<i-l-.5,j-.5<y <j-l-.5}. 



1 Added November 11, 1996, after attending the Visible Human Project Conference 96 in B&- 

thesda, M D. 

2 In general, a little rectangle, but I will normalize to the little square here. The little rectangle 
model is the same mi stake. 
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I have already, with this simple definition, entered the territory of contro- 
versy—a misguided (or at least irrelevant) controversy as I will attempt to show. 
There is typically an argument about whether the pixel "center" lies on the inte- 
gers or the half-integers. The"half-integerists" would have pixel (i, j) correspond 
instead to the area of the plane {(x,y) | 1 <x < <y <j+l.}. 

This model is hidden sometimes under terminology such as the foil owing— 
the case that prompted this memo, in fact: Tlie resolution-independent coordi- 
nate system for an image is -Kx, y) | 0. < x < W/H , 0 .< y < 1.}, W and H are the 
width and height of the image. The resolution dependent coordinate system 
places the edges of the pixels on the integers, their centers on the edges plus one 
half, the upper left corner on (0., 0.), the upper right on (W ., 0.), and the lower left 
on (0., H). See the little squares? Th^ would have edges and centers by this for- 
mulation. 



So What /s a Pixel? 



A pixel is a point sample. It ©<ists only at a point. For a color picture, a pixel 
might actually contain three samples, one for each primary color contributing to 
the picture at the sampling point. Wecan still think of this as a point sample of a 
color. But we cannot think of a pixel as a square— or anything other than a point. 
There are cases where the contributions to a pixel can be modeled, in a low-order 
way, by a little square, but not ever the pixel itself. 

An image is a rectilinear array of point samples (pixels). The marvelous 
SampI i ng Theorem tel Is us that we can reconstruct a conti nuous entity from such 
a discrete entity using an appropriate reconstruction filter^. Figure 1 illustrates 
how an image is reconstructed with a reconstruction filter into a continuous en- 
tity. The filter used here could be, for ©<ample, a truncated Gaussian. To sim- 
plify this image, I use only the footprint of the filter and of the reconstructed pic- 
ture. The footprint is the area under the non-0 parts of the filter or picture. It is 
often convenient to draw the minimal enclosing rectangle for footprints. They are 
simply easier to draw than the footprint— Figure 1(d). I have drawn the minimal 
rectangles as dotted rectangles in Figure!. 



3 And some assumptions about smoothness that we do not need to worry about here. 
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(a) A 5x4 image. 



(b) The footprint of a reconstruction filter. 
A truncated Gaussian, for example. 




Dotted line is minimally enclosing rectangle 



(c) Footprint of image under reconstruction. 



Fixed reference point 




(d) Footprint of reconstructed image. 
Medium quality reconstruction. 




(e) Reconstruction translated (.5, .5), 
then resampled into a 6x5 image. 



Figure 1 
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Fixed reference point 
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(a) A 5x4 image. 




(b) The footprint of a reconstruction filter. 
A cubic, or windowed sine, for example. 




Dotted line is minimally enclosing rectangle 



(c) Footprint of reconstructed image. 
Typical high quality reconstruction. 
Would be resampled into 7x6 image. 



Figure 2 



Figure 2 is tlie same image reconstructed witli a better reconstruction filter— 
eg, a cubic filter or a windowed sine function— and not an unusual one at all. 
M ost qual ity i magi ng uses f i Iters of this vari ety . The i mportant poi nt i s that both 
of these figures illustrate valid image computations. In neither case is the foot- 
print rectangular. In neither case is the pixel ever approximated by a little square. 
If a shape were to be associated with a pixel (and I am not arguing that it should), 
then the most natural thing would betheshapeof the footprint of the reconstruc- 
tion filter. As these two ©<amples show, the filters typically overlap a great deal. 
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Fixed reference point 
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(b) The footprint of a reconstruction filter. 
A simple box filter, for example. 



(a) A 5x4 image. 
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Figure 3 



(c) Footprint of reconstructed image. 
The worst case: low quality reconstruction. 

Figure 3 is tine same image reconstructed witli one of tine poorest reconstruc- 
tion filters— a box filter. The only thing poorer is no reconstruction at all- 
resulting in the abominable "jaggies" of the early days of computer graphics— 
and we will not even further consider this possibility. The Figure 3 reconstruc- 
tion too is a valid image computation, even though it is lacking in quality. This 
lowest-qual ity case is the only one that suggests the I ittle square model . 

So it should be clear that the coordinate system definition given above is not 
suitablefor anything but the lowest-quality imagecomputing.Theedgesof a re- 
constructed image are most naturally defined to be its minimally enclosing rec- 
tangle. But these are dependent on the chosen reconstruction filter. 

The only resolution independent coordinate system that one can safely map 
to an image requires a known reconstruction filter. Given an image, say that rep- 
resented by Figure 2(a), and a known filter, say that of Figure 2(b), then I can de- 
termine exactly what the minimally enclosing rectangle is and map this to the 
normalized rectangle of the proposed definition above: -gx, y) | 0. <x < W/H, 0.< 
y < 1.}. Then the pixel (point sample, remember) locations can be backed out of 
the mapping. Will they sit on the half-integers? In the three cases above, all of 
which are valid, only Figure 3 (the worst) will have the samples on the half- 
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integers under this mapping. Will the left edge of the reconstructed image liedis- 
tance .5 left of the leftmost column of pixels? Again, only in the worst case, Fig- 
ure 3. 

I would suggest at this point that the only thing that is fixed, in general, are 
the samples. Doesn't it make sense that they be mapped to the integers si nee that 
is so simple to do? Then the edges of the reconstructed continuum float depend- 
ing on the chosen filter. I believe that if you rid yourself of the little square 
model, it does not even occur to you to put the samples on the half-integers, an 
awkward position for all but the lowly box filter. The bicubic filter that I most 
often use has a footprint I ike the minimal enclosing rectangle of Figure 2(b). Half- 
integer locations for this filter are simply awkward. Certainly one can do it, but 
why the ©ctra work? 

I believe that the half-integer locations are attractive for "little-squarists" be- 
cause the minpoint (upper left corner) of the reconstructed entity falls at (0, 0). 
But note that this only happens for— yes, again— the box filter. For my favorite 
filter, the minpoint would fall at (-1.5, -1.5). Is that more convenient, prettier, or 
faster than (-2., -2.)? No. 



Why Is the Little square Model So Persistent? 



I believe there are two principal reasons that the little square model hasn't 
simply gone away: 

• Geometry-based computer graphi cs uses it. 

• Video magnification of computer displays appears to show it. 
Geometry-based computer graphics (3D synthesis, CGI, etc.) has solved 

some very difficult problems over the last two decades by assuming that the 
world they model could bedivided into little squares. Rendering is the process of 
converting abstract geometry into viewable pixels that can be displayed on a 
computer screen or written to film or video for display. A modern computer 
graphics model can have millions of polygons contributing to a single image. 
How are all these mi 1 1 ions of geometric things to be resolved into a regular array 
of pi xel s f or d i spl ay? A nswer : Si mpl i fy the probi em by assu mi ng the rectangu I ar 
viewport on the model isdivided regularly into littlesquares, one per final pixel. 
Solve the often-intense hidden surface problem presented by this little square 
part of the viewport. Average the results into a color sample. This is, of course, 
exactly box filtering. And it works, even though it is low order filtering. We 
probably wouldn't be where we are today in computer graphics without this 
simplifying assumption. But, this is no reason to identify the model of geometric 
contributions to a pixel with the pixel. I often meet extremely intelligent and ac- 
complished geometry-based computer graphicists who have leapt to the identifi- 
cation of the little square simplification with the pixel. Thisisnota plea for them 
to desist from use of the little square model. It is a plea for them to be aware of 
the simplification involved and to understand that the other half of computer 
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picturing— the iialf that uses no geometry at all, the imaging half— tries to avoid 
this very simplification for quality reasons. 

When one "magnifies" or "zooms in on" an image in most popular applica- 
tions, each pixel appears to be a little square. The higher the magnification or the 
closer in the zoom, the bigger the little squares get. Since I am apparently magni- 
fying the pixel, it must be a little square, right? No, this is a false perception. 
What is happening when you zoom in is this: Each point sample is being repli- 
cated M xM times, for magnification factor M . When you look at an image con- 
sisting of M xM pixels all of the same col or, guess what you see: A square of that 
solid color! It is not an accurate picture of the pixel below. It is a bunch of pixels 
approximating what you would see if a reconstruction with a box filter were per- 
formed. To do atruezoom requires a resampling operation and is much slower 
than a video card can comfortably support in realtime today. So the plea here is 
to please disregard the squareness of zoomed in "pixels". You are really seeing 
an M xM array of point samples, not a single point sample rendered large. 



How Does a Scanner Digitize a Picture? 



Just to be sure that I eradicate the notion of little square everywhere, let's 
look at the scanning process. I want to be sure that nobody thinks that scanners 
work with little squares and that, therefore, it is all right to use the model. 

A scanner works like this: A light source illuminates a piece of paper con- 
taining a colored picture. Light reflected from the paper is collected and meas- 
ured by color sensitive devices. Is the pictureon the paper divided up into little 
squares, each of which is measured and converted to a single pixel? Not at all. In 
fact, this would be very hard to accomplish physically. What happens instead is 
that the illuminating light source has a shape or the receiving device has an aper- 
ture that gives the incoming light a shape or both, and the de\/ice integrates 
across this shape. The shape is never a square. It is not necessarily accurate, but 
will give you the correct flavor, to think of the shape as a Gaussian. In general, 
overlapping shapes are averaged to get neighboring pixel samples. 

So scanni ng should not contri bute any weight to the I ittle square model . 



How Does a Printer Print a Digital Image? 



A gai n, I et's I ook at a f ami liar process to d etermi ne i f i t contri butes to the I i ttl e 
square model. The process thistimeis printing. This isavery large and compl©< 
subject, because there are many different ways to print an image to a medium. 

Consider printing to film first. There are several ways to do this. In any proc- 
ess where a f I yi ng spot of I i ght i s used to expose the f i I m, then the exposi ng beam 
has a shape— think of it as Gaussian. There are no I ittle squares in this process. 

Consider half-tone printing of ink on paper. The concept here is to convert a 
pixel with many values to a dot of opaque ink on paper such that the area of the 
dot relative to a I ittle square of paper occupied by the dot is in the ratio of the in- 
tensity of the pixel to the maximum possible intensity. Sounds like the little 
square model, doesn't it? But for color printing, the different primaries are 



Microsoft Tech Memo 6 



AIvy 



A Pixel Is Not a Little Square! 8 

printed at an angle to each other so that they can "show through" one another. 
Therefore, although there are little squares in each color separation, there are 
none for the final result 

There is a new technique for printing ink-on-paper that uses stochastic pat- 
terns within each little square. Hence the separations do not have to be rotated 
relative one another in order to "show through". The little square model is a de- 
cent model in this case. 

There are sublimation dye printers and relatives now that print "continuous 
tone" imagesin inkon paper. I bell eve that these use little squares of transparent 
dyes to achieve continuous tone. In that case, they probably use the littlesquare 
model. 

The point here is that the use of the little square is a printing technology de- 
cision, not something inherent in the model of the image being Imaged. In fact, 

the image being imaged is simply an array of point samples in all cases. 



How Is an Image Displayed on a Monitor? 



Many peopleareawarethat a color monitor often has little triads of dots that 
cause the perception of color at normal viewing distances. This is often true, ex- 
cept for Sony Trinitrons that use little strips of rectangles rather than triads of 
dots. In neither case are there little squares on the display. I will assume triads 
for this discussion. It goes through for theTrinitron pattern too, however. 

While we are at it, I would also like to dispel any notion that the triads are 
pixels. There is no fixed mapping between triads and pixels driving them. The 
easiest way to understand this is to consider your own graphics card. Most mod- 
ern cards support a variety of different color resolutions— eg, 640x480, 800x600, 
1024x768, etc. The number of triads on your display screen do not change as you 
change the number of pixels driving them. 

N ow back to the display of pixels on a screen. H ere's roughly what happens. 
The value of a pixel is converted, for each primary color, to a voltage level. This 
stepped voltage is passed through electronics which, by its very nature, rounds 
off the edges of the level steps. The shaped voltage modulates an electron beam 
that is being deflected in raster fashion acrossthefaceof your display. This beam 
has shape— again think of it as Gaussian (although it can get highly distorted 
toward the edges of the display). The shaped beam passes through a shadow 
mask that ensures that only the red gun will illuminatethered phosphors and so 
forth. Then the appropriate phosphors are excited and they emit patterns of light. 
Your eyethen integrates the light pattern from a group of triads into a color. This 
is a complex process that I have presented only sketchily. But I think it is obvious 
that there are no little squares involved at any step of the process. There are, as 
usual in imaging, overlapping shapes that serve as natural reconstruction filters. 

Another display issue that implies the little square model is the notion of 
displays with "non-square pixels". Although it is becoming less common now, it 
used to be fairly common to have a video display driven by, say, a 512x480 im- 
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age memory. This means that 512 samples are used to write a row to the display, 
and there are 480 rows. Video monitors have, if correctly adjusted, a 4:3 aspect 
ratio. So the pixel spacing ratio (PSR) for this case is 1.25 = (4/ 3)/ 512:y 480 = 
(4/ 3)*(480/ 512). So the correct terminology for this case is that the monitor has a 
"non-square pixel spacing ratio", not that it has "non-square pixels". Most mod- 
ern computer displays, if correctly adjusted, have square PSR— ie, PSR =1. 

So we have no contributions from scanni ng or display processes for the little 
square model. We have a case or two for particular printing technologies that 
support a little square model, but they are not the general printing case. In sum- 
mary, the processes used for i mage i nput and output are not sources for the I ittle 
square model. 



What Is a Discrete to Continuous Mapping That Works? 



The only mappi ng that I have been able to come up wi th that doesn't assume 
too much is this: Assume the samples are mapped to the i ntegers (it's so easy af- 
ter all— just an offset to the array indices used to address them in an image com- 
putation). Then the outer extremes of the image are bounded by a rectangle 
whose left edge is (filterwidth/ 2) to the left of the leftmost column of integers, 
right edge is (filterwidth/ 2) to the right of the rightmost column of integers, top 
edge is (filterheight/ 2) above the topmost row of integers, and bottom edge is 
(filterheight/ 2) below the bottommost row of integers occupied by image sam- 
ples). 

This is still not quite complete, because it assumes symmetric filters. There 
are cases— for example, perspective transformations on (reconstructions of) im- 
ages—when asymmetric filters are useful. So the mapping must be generalized 
to handle them. By the way, by symmetrical filter I mean that it is symmetrical 
about its horizontal axis, and it is symmetrical about its vertical axis. I do not 
mean that it is necessarily the same in both dimensions. 

I do believe that it is important to have the vertical dimension increase 
downward. It is not required, but it is so natural that I find it hard to argue 
against. We display from top to bottom on millions of TVs and computer dis- 
plays. We read from top to bottom. We compute on matrices— the natural con- 
tainer for images— from top to bottom. Nearly all popular image file formats 
store images from top to bottom^, or if in tiled form, in rows of tiles from top to 
bottom. 



Imaqe Broadenin 



I mentioned earlier that I would return to a discussion of whether the mini- 
mal enclosing rectangleof a reconstructed imagewerethe most natural represen- 
tation of it. This will fall outof a discussion of image broadening. 



4 TheTarga fileformat is the most flagrant violator of this rule; it also ra/erses RGB to BGR. And 
Windows has adapted theTarga format for its "bitmaps" (without documenting the color chan- 
nel re/ersal, as I unhappily discovered as a developer). 
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By the very nature of the Sampling Theorem that underlies everything that 
we do, during a reconstruction an image suffers broadening due to the width of 
the reconstruction filter. Again, look at Figures 1-3 for examples. The amount of 
broadening is dependent on the filter used. If the filter is asymmetric, then so is 
the broadening. 

I call the excess of a broadened image over itself to be its margin . I n general 
then, an image margin is not symmetric about its image and is dependent on the 
reconstruction filter being used. 

Let's be concrete: Consider scaling 640x480 image down by a factor of 2 in 
both dimensions. The original image is assumed to haveitsminpointat (0,0) and 
its maxpoint (lower right corner) at (639, 479). Assume we are using a symmetric 
bicubic filter for good results. Its canonical footprint is -gx, y) | -2. < x < 2., -2. < y 
< 2.}. Then to do the scaling we reconstruct-transform-resampleS: When we re- 
construct our image into a continuum, it occupies the space from -2. to 641. hori- 
zontally and from -2. to 481. vertically. Thus its minimal enclosing rectangle has 
minpoint (-2., -2.) and maxpoint (641., 481.). If we were to resample at this nno- 
ment, before doing any transform on the reconstruction, we would have broad- 
ened the image by one pixel all around; we would have introduced a 1-pixel 
margi n by the act of reconstruction. (The fi Iter has 0 weight at its extremes so the 
samples along the minimal enclosing rectangle would be 0 and not figure into 
the support of the sampled image.) 

Thisisagood point to pause and notethat it is not necessarily the broadened 
footprint that is interesting. In this case, if we have only a 640x480 display then 
we would crop off the margin pixels for redisplay anyway. In my experience, I 
usually want to know about an image's extent and its margins, before and after 
transformations, in full detail so that I can decide exactly what I want to do. The 
important point is that this information be aval I able, not how. 

Back to the scaling example: Now let's minify the reconstruction by 2. This 
means that the reconstruction is scaled down about its center. This is easily mod- 
eled by scaling down the minimal enclosing rectangle that I will represent as (-2., 
-2.)^(641., 481.). The scaling happens about the center at (319.5, 239.5). There are 
many ways to proceed from here so I will not pursue the details. I believe that I 
have presented enough of the problem and technique of solution that it is clear 
that nothing isoffered to it by a little square model for thepixels. 

In fact, the little square model might have misled us into a compounding of 
low quality techniques: It is tempting to box filter box filters. Thus it is tempting 
to take the 640x480 example above and look at each set of 2x2 pixels to scale 
down by 2. It is another application of the box filter to simply average each 2x2 



5 Generally, there needs to be a low-pass filtering step too, before resampling. When scaling up, 
no high frequencies are introduced, but when scaling down, the/ are and must be low-pass fil- 
tered to sat! sfy the Samp! 1 ng Theorem. 
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set of pixels into a single new pixel. This is generally a bad idea and not the path 
to take for good results. 



Summar 



I have presented a brief but inclusive analysis of sampling and filtering. It 
has been shown that the little square model does not arise naturally in sampling 
theory, the main underpinning of everything we do. It does not arise in scanning 
or display. It arises in printing only in restricted cases. The geometry world uses 
it a great deal because th^ have had to simplify in order to accomplish. Their 
simplified model of contributions to a pixel should not be confused with or identi- 
fied with the pixel. Magnified screen pixels that look like little squares have been 
shown to bea quick and dirty trick (pixel replication) by graphics boards design- 
ers, but not the truth. In short, the little square model should be suspect when- 
ever it is encountered. It should be used with great care, if at all, and certainly 
not offered to the world as "standard" in image computing. 

In the process cf this presentation, I have demonstrated at least one very 
natural mapping of the discrete samples of an image to a continuum recon- 
structed from it. The method used is straightforward and always works. I have 
also argued that the following details of such a reconstruction are interesting 
enough to track: the ©<act filter used, the image margins created by reconstruc- 
tion, and minimal enclosing rectangles. I havealso suggested that it is natural for 
the vertical coordinate to increase downwards, and that it is simple and natural 
for sample locations to be mapped to the integers. However, neither of these is 
required, so long as consistency reigns. I do argue, however, that placing sam- 
ples at the hal f-i ntegers seems to i nd i cate a I u rki ng rel u ctance to d i smi ss the I i ttl e 
square model, or the box filter, the only two common situations where the half- 
integers are natural concepts. 

Finally, I have pointed out two related misconceptions: (1) The triads on a 
display screen are not pixels; they do not even map one-to-one to pixels. (2) Peo- 
ple who refer to displays with non-square pixels should refer instead to non- 
square, or non-uniform, pixel spacing. 
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